Skip to content

perf(ci): disable the Turbopack persistent build cache (3.2x faster builds) - #6080

Merged
waleedlatif1 merged 2 commits into
stagingfrom
fix/disable-turbopack-build-cache
Jul 30, 2026
Merged

perf(ci): disable the Turbopack persistent build cache (3.2x faster builds)#6080
waleedlatif1 merged 2 commits into
stagingfrom
fix/disable-turbopack-build-cache

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

The Turbopack persistent build cache is a net loss at this app's size. Turning it off cuts the Build App check from 8–20 min to under 3.

Measured, not inferred — a controlled A/B in #6078: three runs on one branch, every commit touching only .github/**, so the Next module graph was byte-identical across all three and only cache state varied.

run flag pre-build disk compile Build App job
C — cache OFF 5.1G (ignored) 113s 2m53s
A — cache ON, cold 4.0K 162s 3m54s
B — cache ON, warm 5.1G 360s 8m18s

A warm cache made the identical build 3.2× slower than no cache.

Why it degrades: the cache grows, it doesn't converge

Run B started from 5.1 GB and committed 12 GB — on a tree where nothing changed. That's the mechanism behind the progressive slowdown on longer-lived disks (observed up to 11.7 min): every run adds to what the next must read and revalidate, so a disk gets worse the more it's used. This also corrects an earlier theory of mine that cache mismatch was the variable — accumulated size matters more.

Validity controls

  • Flag manipulation confirmed in the logs — cache-on runs print ✓ turbopackFileSystemCacheForBuild; the cache-off run omits it entirely.
  • Every run used turbo --force. Without it, a .github-only commit leaves Turbo's task inputs unchanged, so Turbo replays a cached log and reprints a stale compile time — fabricating the number being measured. Five such phantom runs had to be discarded from the observational data first.
  • No cancelled siblings. cancel-in-progress means an early push cancels a run, and a cancelled run still commits its partial cache. Each run here reached completed before the next push. This mattered: an earlier conclusion of mine ("a brand-new sticky key is not cold") was wrong precisely because the "first" run had a cancelled sibling that had already written 5.1 GB.

What this removes, and what it doesn't

#5869 enabled the cache on locally-measured numbers (105s cold → 22s warm) that never reproduced in CI and are inverted here. #6072 then branch-scoped the disk to stop PRs restoring each other's caches — that fixed a real problem, but with the cache off the disk serves nothing, so the mount, the size reporting, and the env gate go with it.

turbopackFileSystemCacheForBuild: false is pinned explicitly rather than left to the Next default, because upstream already flips that default to true in canary/preview builds (#94616) — relying on the default would let a version bump silently re-enable a config we measured as harmful.

ci-cache-cleanup.yml is kept, re-scoped. PRs opened while the per-branch key was live still hold 5–12 GB volumes and nothing else reclaims them; deleting the workflow now would orphan that storage. It's a no-op for new PRs and can be removed once drained.

Caveat

n=1 per cell. The 3.2× effect size and its agreement with ~15 prior observations make this convincing, but it is three runs, not a distribution. The change is a one-line revert if staging disagrees.

Cross-commit restore is separately undocumented-as-supported (#87283 reports stale HTML from a cache built at another commit), so turning it off also removes a correctness risk.

Type of Change

  • Performance
  • Improvement

Testing

tsc --noEmit exit 0, biome clean, actionlint clean on both changed workflows. No dangling references to NEXT_TURBOPACK_BUILD_CACHE or the nextjs-cache key remain. This PR's own Build App run is the confirmation — expect ~2–3 min.

It is a net loss at this app's size. A controlled A/B on one branch (#6078),
three runs with a byte-identical module graph so only cache state varied:

  cache OFF        113s compile, 2m53s job
  cache ON, cold   162s compile, 3m54s job
  cache ON, warm   360s compile, 8m18s job

The cache made the same build 3.2x slower. It also grew 5.1 GB -> 12 GB
across two runs of an unchanged tree, which explains the progressive
degradation seen on longer-lived disks (up to 11.7 min): the more a disk is
written, the more the next run must read and revalidate. Flag manipulation
is visible in the logs — the cache-on runs print
`✓ turbopackFileSystemCacheForBuild`, the cache-off run omits it — and every
run used `turbo --force` so none is a replayed log.

#5869 enabled this on locally-measured numbers (105s cold -> 22s warm) that
never reproduced in CI and are inverted here. #6072 then branch-scoped the
disk to stop PRs restoring each other's caches; that fixed a real problem,
but with the cache off the disk is unnecessary, so the mount, the pre/post
size reporting, and the env gate all go with it.

Pins `turbopackFileSystemCacheForBuild: false` explicitly rather than relying
on the Next default: upstream already flips that default to true in
canary/preview builds (vercel/next.js#94616), so leaning on the default would
let a version bump silently re-enable this.

Keeps ci-cache-cleanup.yml, re-scoped to draining the 5-12 GB volumes that
PRs opened while the per-branch key was live still hold — nothing else
reclaims them. It is a no-op for new PRs and can be deleted once drained.

Caveat: n=1 per cell. The 3.2x effect size and agreement with ~15 prior
observations make it convincing, but this is three runs, not a distribution.
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Jul 30, 2026 2:02am

Request Review

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
CI and Next build-cache config only; no app runtime, auth, or data paths. Reversible if staging build times disagree.

Overview
Disables Turbopack’s persistent build cache after A/B showed it slowed identical CI compiles (~113s off vs ~360s warm) and bloated disk (5→12 GB). turbopackFileSystemCacheForBuild is hard-coded to false in next.config.ts so a Next upgrade can’t silently turn it back on.

CI build job drops the .next/cache sticky disk mount, NEXT_TURBOPACK_BUILD_CACHE, and pre/post cache size reporting; comments document why the mount is gone.

ci-cache-cleanup.yml stays only to delete legacy per-PR sticky disks from when the cache was mounted; it’s documented as temporary drain work, not tied to the removed mount step.

Reviewed by Cursor Bugbot for commit e539058. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Disables Turbopack’s persistent build cache and drops the CI .next/cache sticky-disk mount after measured 3.2× slower warm builds.

  • Pins turbopackFileSystemCacheForBuild: false in apps/sim/next.config.ts (no longer gated by NEXT_TURBOPACK_BUILD_CACHE).
  • Removes the Next.js cache mount and pre/post size reporting from test-build.yml.
  • Keeps ci-cache-cleanup.yml as a legacy drain for per-branch disks from older PRs, with updated comments.

Confidence Score: 5/5

This PR appears safe to merge; no blocking failures remain.

Prior stale cleanup-comment feedback is addressed on HEAD, and the cache-off plus mount-removal path leaves no remaining blocking failure.

Important Files Changed

Filename Overview
apps/sim/next.config.ts Hard-disables turbopackFileSystemCacheForBuild so CI/prod cannot re-enable the cache via env or Next default flips.
.github/workflows/test-build.yml Drops Next.js sticky-disk mount, cache size reports, and NEXT_TURBOPACK_BUILD_CACHE from the Build App job.
.github/workflows/ci-cache-cleanup.yml Documents legacy-only drain of old nextjs-cache disks; delete-key pattern unchanged for backlog reclamation.

Reviews (2): Last reviewed commit: "docs(ci): correct the cleanup key commen..." | Re-trigger Greptile

Greptile P2: the delete step still claimed its key must stay byte-identical
to the Mount Next.js build cache step in test-build.yml, but this PR removes
that mount. It is now a hard-coded legacy drain key that mirrors nothing.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

Fixed the P2 in ci-cache-cleanup.yml (e539058). You were right — the step comment claimed its key had to stay byte-identical to the Mount Next.js build cache step that this PR removes, so it referenced something that no longer exists.

It now states what the key actually is: a hard-coded legacy drain key that mirrors nothing, with an explicit warning not to retarget or delete it while pre-removal PRs are still open (their 5–12 GB disks would otherwise never be reclaimed).

Flagging it as an outside-diff comment rather than a thread was the right call given the line wasn't in the diff — but it also meant reviewThreads reported zero unresolved, so the 4/5 score was the only signal it existed.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit e539058. Configure here.

@waleedlatif1
waleedlatif1 merged commit 3e62546 into staging Jul 30, 2026
27 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/disable-turbopack-build-cache branch July 30, 2026 02:06
waleedlatif1 added a commit that referenced this pull request Aug 1, 2026
…restarts) (#6151)

* perf(dev): re-enable the Turbopack dev filesystem cache (5.4x faster restarts)

`turbopackFileSystemCacheForDev` has been `false` since #5408 — a landing-page
homepage redesign whose description covers hero cards, feature-card aspect
ratios, eyebrow chips and a voice-input button color, and never mentions
Turbopack, caching, or dev performance. It was collateral, not a decision, and
it overrode the Next default (true since v16.1).

It is not the flag #6078/#6080 measured. That A/B was `...ForBuild` and its
conclusion stands — the build cache is a 3.2x regression and stays off. The two
flags look alike and are opposite decisions; both are now commented as such.

Measured on `/workspace/[workspaceId]/w`, n=3 per arm, SIGINT between runs:

  cache OFF   31.4s / 30.1s / 31.9s   RSS 9.0-9.8 GB
  cache ON     5.6s /  5.6s /  5.5s   RSS 4.4-5.1 GB

5.4x faster restarts, ~2x less resident memory. Cold compile against an empty
cache is unchanged (~32s either way) — the cache only pays back on restart,
which is the loop that actually hurts.

The cache is unbounded on disk: the abandoned one on this machine had reached
78 GB across 1,848 SST files, and a stale cache is slower to read back, so left
alone it erodes the win it exists to provide. `prune-turbopack-cache.ts` runs on
`predev` and drops it past a cap (default 20 GB, `SIM_TURBOPACK_CACHE_MAX_GB` to
override); `bun run dev:cache:prune` forces it. It never blocks `next dev` on a
maintenance failure.

Adds a `dev-performance` skill recording the cost model, the reference numbers,
and the benchmarking method — including that stopping the server with `kill -9`
mid-cache-write discards the cache and makes this exact win read as no win.

* improvement(dev): chain the cache prune into dev scripts instead of a predev hook

Review read the root `bun run dev` path as bypassing the `predev` hook and so
never capping the newly-enabled cache. Turbo does fire `pre*` hooks — verified
live, the run prints the prune before `next dev` — but the concern is fair in
that the guarantee rested on package-manager lifecycle semantics that are
invisible at the call site.

Chaining it explicitly removes the question entirely: every `dev` variant now
runs `bun run dev:cache:cap && …`, which holds on any invocation path, is
visible in the command itself, and drops the three duplicated `predev:*` entries
for one shared script.

Verified on both paths — direct `bun run dev` and root `turbo run dev`, the
latter printing:

  sim:dev: $ bun run dev:cache:cap && next dev --port 3000
  sim:dev: $ bun run ../../scripts/prune-turbopack-cache.ts

* docs(dev): document cache-corruption recovery, the cost of enabling the cache

Stress-tested the failure mode rather than assuming it: deliberately corrupting
an SST block makes Turbopack abort with a FATAL panic — it does not self-heal.

  FATAL: An unexpected Turbopack error occurred.
  Cache corruption detected: checksum mismatch in block 4 of 00000221.sst

`bun run dev:cache:prune` and restart fixes it; verified the canvas serves 200
again afterwards. Documented in the skill and in the script's header, since the
symptom is a hard crash and the remedy is not guessable.

This is the honest cost of turning the cache on. It is worth paying — a 5.4x
faster restart against a rare, loud, single-command failure — but it should be
written down rather than discovered.

Worth distinguishing from the adjacent case: an ordinary hard kill does *not*
corrupt the cache. Turbopack discards a partially-written cache and rebuilds it
silently, which is exactly why a `kill -9`-based benchmark reads as "no cache
win" (noted in the benchmarking section).

* refactor(dev): drop the dev-performance skill, keep its findings at the code

A whole skill was too much for what this is. The parts that are load-bearing —
why the two lookalike cache flags are opposite decisions, the measured numbers,
the corruption remedy, and the benchmarking trap — now live in the config and
script they describe, where someone changing the flag actually reads them.

The trap is the piece worth keeping: `next dev` compiles on demand so startup
time is meaningless, and stopping the server with `kill -9` makes Turbopack
discard a partially-written cache and rebuild silently — which reads as 'the
cache does nothing' and is how this flag stayed wrong for a month.

Dropped rather than relocated: generic advice that was not specific to this repo
(antivirus, Docker-on-macOS, orphaned processes) and a measured no-op
(`optimizePackageImports` for lucide-react changed nothing, 31.6s vs 31.7s).

* docs(dev): record the measured cost and concurrency behaviour of cache pruning

Stress-tested the maintenance path rather than assuming it is free.

Cost: the size walk is ~30ms on a real cache and ~85ms at 2,000 files — under 2%
of a 4.2s warm restart, and invisible against a cold one. It runs before every
dev start, so it needed to be cheap; it is.

Concurrency: pruning while a dev server is live (which happens when a second
server is started from the same checkout) does not crash it. The running server
keeps its in-memory state and kept serving HTTP 200 with zero panics. It does
stop persisting for the rest of that session, so its next start is cold once —
verified recovering at 23.4s then 4.5s. Worth writing down because the directory
silently never reappears mid-session, which looks like a bug if you go looking.

The cap is a backstop, not routine: a normal session sits at 1-2 GB against a
20 GB default.

* fix(dev): cap every app's Turbopack cache, not just apps/sim

`apps/docs` is a Next app too (`next dev --port 3001`) and overrides nothing, so
it uses the Next default where the dev filesystem cache is on. It already had an
uncapped 1.1 GB cache here, and the root `bun run dev` (`turbo run dev`) starts
it — so a teammate using the documented command was accumulating a cache nothing
would ever prune.

The script now resolves its target from the working directory instead of
hardcoding `apps/sim`, and each app chains its own cap. Per-app rather than one
sweep on purpose: a single pass would let one app's dev start delete a cache
another app is holding open, which costs that session its persistence.

Verified both: `apps/sim` and `apps/docs` each report and cap their own 1.1 GB
cache, and both dev servers start clean (`Ready in 299ms` / `229ms`, docs serving).

* refactor(dev): drop dev:cache:prune in favour of the existing dev:clean

`dev:cache:prune` duplicated `dev:clean`, which already existed in `apps/sim` and
does strictly more (`rm -rf .next/dev/cache` covers the Turbopack cache plus the
fetch and image caches). Two commands for one job is worse than one, and the
docs pointed at the newer, narrower of the two.

Removes it from both apps and gives `apps/docs` the `dev:clean` that `apps/sim`
already had, so the recovery command is the same everywhere. `dev:cache:cap`
stays — it is the chained step, used by more than one dev variant, and naming it
keeps the relative script path out of each command.

Verified `dev:clean` is a real remedy: corrupt a cache block, run it, restart —
canvas serves 200 with no panic.

Also corrects an overstatement. A damaged cache does not *always* abort
Turbopack; whether it panics depends on whether the damaged region is read, so
it is not reliably reproducible. Both notes now say "can abort" and give the same
remedy either way.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant